Remote Direct Memory Access (RDMA) is a direct memory access mechanism that enables a computer to access memory from another computer without involving the computers' operating systems. RDMA supports zero-copy networking by enabling a network adapter to transfer data directly to or from application memory, eliminating the need to copy data between application memory and the data buffers in the operating system. Such transfers require no work to be done by CPUs, caches, or context switches, and transfers continue in parallel with other system operations. When an application performs an RDMA Read or Write request, the application data is delivered directly to the network, reducing latency and enabling fast message transfer.
To efficiently communicate with remote systems via user space (i.e., the non-kernel memory space allocated for applications by an operating system), conventional RDMA devices require pre-registered, pre-pinned memory regions for all data transfers over the fabric or network. This consumes large amounts of system memory that could be used by other applications. In order to avoid page faults, memory may often be overallocated to (hopefully) address worst-case traffic conditions. However, under heavy traffic loads even this approach may fail, leading to page faults under which the amount of memory allocated to a pre-pinned memory region is insufficient, resulting in temporary use of virtual memory that is accessed from local or remote secondary storage devices rather than system memory; these devices, such as hard disk drives, have access speeds that are an order of magnitude or more slower than typical system memory. Under conventional approaches, page faults are either transparent to RDMA senders or are otherwise identified indirectly well after the page fault has occurred (e.g., lack of ACKnowledgements within a timeout period may indicate some type of fault or congestion).
There are several RDMA capable network interface cards (RNIC) available on today's market that provide both open source and proprietary methods for implicit memory registration. They all attempt to remove the requirement of pre-pinning memory regions for RDMA transfers. In these cases, the RNIC essentially acts as a memory management unit (MMU) and provides some form of synchronization with system MMU. This MMU synchronization comes in many forms but essentially guarantees that the adapter will participate in all user memory region accesses and tolerate a page fault and page pinning during data transfers. These paging events are indeterminist and can stall the data stream significantly, especially if the system is busy or if the fault requires paging from a local or network attached drive.